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Introduction 

The  impetus  behind  the  national  ball  plate  round  robin,  administrated  by  the  National  Institute 
of  Standards  and  Technology  (NIST)  in  cooperation  with  the  National  Conference  of  Standards 
Laboratories  (NCSL)  and  the  University  of  North  Carolina  at  Charlotte  (UNCC),  was  to  provide 
a simple  method  for  the  assessment  of  the  current  state  of  industrial  measurement  capability  using 
coordinate  measuring  machines  (CMMs).  Historically,  round  robins  (where  a single  artifact  is 
circulated  among  the  participants  for  measurement)  have  been  well  suited  for  such  a purpose. 
In  particular,  this  round  robin  was  modelled  after  the  CIRP  (International  Institution  for 
Production  Engineering  Research)  international  round  robin  for  which  a similar  ball  plate  was 
sent  to  various  national  laboratories  for  measurement.  This  paper  presents  the  interim  results  of 
the  NIST  round  robin  along  with  some  analysis  of  these  results. 


Round  Robin  Demographics 

The  round  robin  participants  included  various  US  public  and  private  manufacturing  institutions 
that  are  engaged  in  coordinate  metrology  using  coordinate  measuring  machines.  In  order  to 
provide  a fair  comparison,  only  computer  controlled  coordinate  measuring  machines  were 
included  in  the  study.  Since  the  goal  of  this  round  robin  was  to  assess  industrial  measurement 
practices,  manufacturers  of  CMMs  were  also  excluded  because  they  have  the  advantage  of 
routinely  performing  extensive  characterizations  of  their  machines. 

There  were  a total  of  16  organizations  that  volunteered  to  participate  in  this  round  robin, 
representing  a substantial  portion  of  the  manufacturing  spectrum.  Most  participants  are  leaders 
in  their  fields,  which  include  aerospace,  heavy  equipment,  petroleum  equipment  and  defense 
facilities.  To  date,  13  participants  have  reported  results  of  ball  plate  measurements  on  a total  34 
CMMs.  This  sampling  of  machines  represented  most  of  the  US  and  foreign  coordinate  measuring 
machine  manufacturers  and  ranged  from  ultra  precision  to  shop  floor  accuracy  machines  with  axis 
lengths  up  to  2.5  meters. 
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Ball  Plate  Description 


The  ball  plate  used  for  this  round  robin,  shown  in  figure  1,  is  a two-dimensional  array  of  tool 
steel  balls  mounted  on  a base  plate.  The  plate  consists  of  16  spheres  on  an  equally  spaced  grid 
with  a nominal  spacing  of  80  mm.  The  nominal  coordinates  of  these  spheres  and  the  distance 
from  the  plate  coordinate  system  origin  are  given  in  table  1.  The  spheres  are  25.4  mm  (1  inch) 
in  diameter  and  are  round  to  better  than  0.15  micrometer.  The  base  plate  is  constructed  from 
mild  carbon  steel  with  three  semi-kinematic  pads  ground  on  the  bottom  for  mounting.  It  is 
important  that  the  artifact  remain  dimensionally  stable  for  the  duration  of  the  round  robin.  Hence 
in  order  to  avoid  damage  to  the  plate  (which  could  change  its  measurement  values),  the  size  of 
the  ball  plate  was  limited  to  0.24  m by  0.24  m,  and  the  plate  was  made  quite  massive  by  having 
a thickness  of  25.4  mm  (1  inch).  Upon  completion  of  the  round  robin  the  ball  plate  will  be 
remeasured  by  NIST  to  confirm  its  dimensional  integrity. 


Figure  1 . The  ball  plate  used  in  the  round  robin.  The  spheres  are  numbered  sequentially  from 
left  to  right  beginning  with  number  1 in  the  lower  left  comer 


The  ball  plate  was  chosen  as  the  measurement  artifact  for  this  round  robin  due  to  its  simple 
design  and  ease  of  measurement.  Most  coordinate  measuring  machine  users  are  familiar  with 
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the  measurement  of  spheres  due  to  their  widespread  use  as  probe  calibration  artifacts.  Although 
a more  complicated  artifact  could  have  been  chosen  to  better  represent  manufactured  parts,  it  was 
felt  that  this  type  of  artifact  was  a suitable  choice  for  the  broad  spectrum  of  round  robin 
participants.  We  point  out  that  ball  plate  measurements  provide  only  limited  information  about 
the  CMM.  In  particular,  since  this  plate  was  designed  to  be  measured  in  the  plane  of  the  CMM 
table  no  information  is  determined  regarding  the  vertical  axis  of  the  CMM,  e.g.,  Z axis 
squareness  or  Z axis  roll  etc.  Furthermore,  since  only  the  center  to  center  distances  between  the 
spheres  are  reported,  many  effects  such  as  probe  lobing  or  repeatability  problems  which  occur 
at  each  probing  point  are  averaged  out  in  the  sphere  fitting  process.  Finally  the  center  to  center 
distances  are  insensitive  to  errors  in  the  effective  size  of  the  stylus,  which  will  appear  in  actual 
size  measurements  such  as  the  length  of  a gage  block.  Consequently,  the  ball  plate  results  should 
be  viewed  as  representing  only  a fraction  of  the  potential  errors  which  could  occur  in  an  actual 
part  measurement. 


Table  1.  - Nominal  Sphere  Coordinates  and  Distances  from  Ball  #1. 


Ball  Number 

X Coordinate 
(mm) 

Y Coordinate 
(mm) 

Distance  from 

Ball  Number  1 
(mm) 

1 

0 

0 

0 

2 

80 

0 

80 

3 

160 

0 

160 

4 

240 

0 

240 

5 

0 

80 

80 

6 

80 

80 

113 

7 

160 

80 

178 

8 

240 

80 

252 

9 

0 

160 

160 

10 

80 

160 

178 

11 

160 

160 

226 

12 

240 

160 

288 

13 

0 

240 

240 

14 

80 

240 

252 

15 

160 

240 

288 

16 

240 

240 

339 
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Measurement  Procedures 


The  participants  were  asked  to  measure  the  plate  and  report  the  X and  Y ball  center  coordinates 
along  with  the  estimated  errors  in  these  coordinates.  The  participants  were  given  the  option  of 
measuring  the  plate  on  multiple  machines  (if  applicable)  as  long  as  each  measurement  was 
independently  reported.  The  only  instructions  pertained  to  the  reporting  of  the  data  which  stated 
that  the  center  of  ball  number  one  was  to  be  defined  as  the  origin  of  the  plate  coordinate  system, 
that  the  center  of  ball  number  four  was  to  be  defined  to  be  on  the  plate  X axis,  and  that  the  X 
and  Y ball  center  coordinates  were  to  be  reported  referenced  at  standard  temperature  20  °C  (the 
nominal  coefficient  of  thermal  expansion  was  supplied  for  this  plate  as  11.6  ppm/°C).  The  latter 
requirement  was  left  to  the  discretion  of  the  individual  participants  because  correction  for  the 
thermal  expansion  of  the  part  and  the  CMM  scales  is  part  of  the  measurement  plan  and  is 
therefore  included  in  the  overall  assessment.  There  were  no  other  instmctions  on  how  the  plate 
was  to  be  measured  - simply  that  the  plate  be  treated  as  a precision  part.  The  measurement  plan, 
such  as  the  number  of  points  per  sphere,  number  of  times  each  sphere  was  to  be  measured,  the 
location  and  orientation  of  the  plate  measurement(s)  within  the  machine  measurement  volume, 
probing  parameters  (probe  approach  rate,  probe  approach  distance),  etc.,  were  also  left  to  the 
discretion  of  the  individual  participants. 


Results  and  Data  Analysis 

The  coordinate  deviations  reported  in  this  paper  are  with  respect  to  the  mean  of  baseline 
measurements  made  by  NIST  and  UNCC  on  their  respective  coordinate  measuring  machines. 
These  results  were  chosen  as  the  calibrated  ball  center  coordinates  based  on  an  intercomparison 
between  the  two  institutions.  The  worst  case  difference  from  the  mean  of  the  baseline 
measurements  was  0.5  micrometer  with  typical  agreement  better  than  0.25  micrometer.  These 
measurements  were  made  on  two  completely  different  standard  production  accuracy  coordinate 
measuring  machines.  Both  organizations  used  redundant  measurement  techniques  [1,2]  which 
effectively  eliminate  most  of  the  residual  CMM  errors  which  would  affect  the  measurement 
results  for  this  ball  plate. 

The  results  of  the  ball  plate  round  robin  are  shown  below  in  figures  2 through  6.  Figure  2 shows 
the  deviations  in  distance  from  ball  number  1 for  all  of  the  participants.  It  is  also  instructive  to 
view  the  distance  deviations  as  parts  per  million  (ppm)  of  error.  Figure  3 is  a plot  of  this  data 
as  a function  of  distance  from  ball  number  1 . This  gives  an  indication  of  how  well  the  individual 
machines  could  measure  two-dimensional  features  of  various  lengths.  The  majority  of  the 
machines  are  within  ± 50  ppm  which  translates  to  ± 50  micrometers  per  meter  of  feature  length. 
However,  it  is  apparent  from  this  data  that  over  short  distances  some  of  the  machines  could  not 
do  better  than  100  micrometers  per  meter. 
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Figure  2.  Distance  deviations  from  ball  # 1 
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Figure  3.  The  same  distance  deviations  as  in  figure  2,  expressed  on  a part  per  million  basis 
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Data  reporting  for  this  round  robin  was  to  include  an  estimate  of  the  measurement  error  stated 
as  three  (3)  standard  deviations  plus  any  estimated  systematic  errors.^  Providing  this  error 
estimate  proved  to  be  difficult  for  many  of  the  participants,  and  varied  from  the  estimates  as 
requested  to  no  estimates  at  all.  Figure  4 shows  the  error  estimates  (±)  plotted  around  the 
maximum  coordinate  deviation,  from  the  baseline  coordinates,  for  each  participant.  These 
deviations  were  chosen  without  regard  to  ball  number  or  coordinate  (X  or  Y). 


Figure  4.  Measurement  deviations  and  error  estimates.  The  deviations  are  represented  by  the 
solid  circles  and  the  error  estimates  by  the  error  bars.  Note  that  there  were  several 
participants  who  did  not  provide  an  error  estimate. 

It  can  be  seen  from  this  plot  that  the  maximum  deviations  for  a majority  of  the  participants  are 
within  ± 5 micrometers  (±  0.0002  inch)  of  the  baseline  coordinates.  Using  a typical  gaging  ratio 
of  10:1  (i.e.,  the  "gage  maker's  rule")  or  4:1  (as  suggested  by  MIL  STD  45662A)  of  part 
tolerance  to  measurement  error  indicates  that  most  participates  can  inspect  parts  with  tolerances 
as  tight  as  ± 50  micrometers  (±  0.002  inch)  using  the  10:1  rule  or  ± 20  micrometers  (±  0.0008 
inch)  using  the  4:1  rule.  However,  as  previously  discussed,  there  may  be  additional  errors  not 
detected  by  the  ball  plate  results  which  further  degrade  CMM  performance.  Perhaps  more 
disturbing  is  that  over  90  percent  of  the  participants  failed  to  correctly  estimate  the  measurement 


' "Measurement  error"  is  different  from  "expanded  uncertainty"  which,  as  defined  by  NIST  policy  [3], 
usually  reports  two  standard  deviations.  This  method  was  used  to  be  consistent  with  previous  CIRP 
round  robin  results. 
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error  on  at  least  one  of  the  balls  on  the  plate.  (Figure  4 does  not  necessarily  represent  the  worst 
case  result  regarding  error  estimation  because  the  maximum  deviation  and  the  poorest  error 
estimates  need  not  be  the  same.)  This  strongly  suggests  that  realistic  CMM  error  estimation 
techniques  must  be  developed  by  the  CMM  community  as  a whole. 

Ball  plates  have  historically  been  used  for  the  periodic  performance  evaluation  and/or  calibration 
of  coordinate  measuring  machines  [2,4].  The  amount  of  information  about  the  CMM  that  may 
be  obtained  from  measurements  of  this  type  of  artifact  depends  on  the  number  and  location  of 
ball  plate  measurements,  the  suitability  of  the  plate  to  be  mounted  in  horizontal  and  vertical 
planes  (as  well  as  at  oblique  angles)  and  whether  the  ball  plate  is  calibrated. 

Without  imposing  a rigorous  and  well  planned  measurement  strategy  on  the  participants,  it  would 
be  difficult  to  deconvolve  all  of  the  individual  errors  for  a series  of  independent  ball  plate 
measurements  such  as  this  round  robin  data.  This  is  due  to  the  highly  interdependent  nature  of 
the  these  errors.  However,  we  may  examine  the  individual  X and  Y coordinate  deviations 
looking  for  structure  in  the  data.  As  can  be  seen  from  figures  5 and  6,  much  of  this  data  display 
a systematic  structure,  and  in  some  cases,  it  is  possible  to  identify  one  or  more  of  the  dominant 
errors  from  this  data.  For  instance,  by  comparing  the  X and  Y deviations  for  a single  set  of 
measurements  it  is  possible  to  identify  relative  scale  errors  and/or  thermal  errors.  By  rearranging 
the  X and  Y deviations  in  groups  of  nominally  equal  Y and  X coordinates,  respectively,  these 
errors  become  apparent. 
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Figure  5.  X coordinate  deviations 
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Figure  6.  Y coordinate  deviations 


For  example,  if  the  deviations  of  the  X and  Y individual  ball  coordinates  are  nominally  equal, 
as  in  figure  7,  then  a thermal  error  is  suspect.  This  type  of  error  could  result  from  measurements 
made  at  a temperature  different  from  20  °C  or  from  improper  correction  for  making  such 
measurements  due  to  the  lack  of  accurate  temperature  and/or  accurate  coefficient  of  thermal 
expansion  (CMM  scales  and/or  plate)  data.  It  may  also  be  an  artifact  of  the  machine  scales 
resulting  from  an  equal  scale  error  or  scales  compensated  at  a temperature  different  from  20  °C. 

Another  identifiable  error  is  the  out-of-squareness  between  the  two  measurement  axes.  Since  the 
X axis  of  the  ball  plate  was  defined  by  the  plate  coordinate  system,  any  squareness  error  would 
appear  as  an  out-of-squareness  of  the  Y axis  with  respect  to  the  X axis.  (The  assignment  of  out- 
of-squareness  to  either  of  the  axes  is  purely  arbitrary  as  long  as  the  magnitude,  direction  and 
effect  on  the  measured  coordinates  is  understood.)  Therefore,  by  reordering  the  X deviation  data 
in  groups  of  nominally  equal  X,  any  out-of-squareness  between  the  X and  Y axes  will  become 
evident.  Figure  8 shows  evidence  of  a classical  out-of-squareness  error  for  one  participant's 
CMM.  In  this  case  the  out-of-squareness  is  calculated  as  45  microradians  (approximately  9 arc 
seconds)  of  error. 

Although  we  have  presented  the  results  from  particular  CMMs  to  illustrate  these  errors,  they  were 
frequently  found  among  the  participants.  The  presence  of  such  systematic  errors  indicates  that 
more  frequent  testing  of  CMMs,  i.e.  statistical  process  control,  maybe  needed. 
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• X Coordinate  n Y Coordinate  X Best-Fit  - - Y Best— Fit 

Figure  7.  An  example  of  thermal  error  for  one  participant's  machine 


Figure  8.  One  participant's  data  showing  an  out-of-squareness  error;  in  this  case  the  error  is  45 
microradians  (approximately  9 arcseconds) 
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Summary 


We  have  presented  the  available  results  of  the  NIST/NCSL/UNCC  ball  plate  round  robin  which 
is  currently  in  progress.  Data  for  34  coordinate  measuring  machines,  from  13  different 
organizations,  have  thus  far  been  analyzed.  The  ball  plate  results  are  indicative  of  the  level  of 
precision  that  might  be  obtained  on  small  (<  250  mm)  precision  parts.  Typical  results  were 
within  5 micrometers  of  the  baseline  measurements;  however,  errors  exceeding  20  micrometers 
were  present.  Expressed  on  a parts  per  million  basis,  the  majority  of  these  deviations  were  less 
than  25  ppm,  but  in  some  cases  exceeded  100  ppm.  A brief  discussion  of  the  data  presented 
some  of  the  possible  sources  of  error  that  contributed  to  these  deviations.  Examples  of 
temperature  and  squareness  errors,  taken  from  actual  round  robin  data,  were  described.  Of  the 
participants  that  estimated  their  measurement  errors,  over  90  percent  exceeded  this  estimate  on 
one  or  more  of  the  spheres.  This  suggests  that  a higher  level  of  measurement  process  control, 
e.g.,  regular  interim  testing  of  coordinate  measuring  machines  may  be  needed  to  provide 
statistical  process  control  for  industrial  measurement  practices. 
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